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A  Psychometrically  Sound  Cognitive  Diagnostic  Model: 

Effect  of  Remediation  as  Empirical  Validity 

Abstract 

The  purpose  of  this  study  was  to  validate  the  results  of  cognitive 
diagnoses  using  the  rule -space  model  and  to  demonstrate  the  usefulness 
of  cognitive  diagnoses  for  instruction.  The  results  of  the  study 
strongly  indicated  that  the  rule -space  model  can  effectively  diagnose 
students'  knowledge  states  and  can  point  out  ways  for  remediating  their 
errors  quickly  with  minimum  effort. 

It  was  also  found  that  the  designing  of  instructional  units  for 
remediation  can  be  effectively  guided  by  the  rule-space  model,  because 
the  determination  of  all  possible  ideal  item- score  patterns,  given  an 
incidence  matrix,  is  based  on  a  tree  structure  of  cognitive  attributes, 
knowledge  states ,  and  items . 


Introduction 


The  diagnosing  of  cognitive  errors  committed  by  students  taking  a  test 
is  a  matter  of  interest  not  only  to  teachers,  but  also  to  cognitive 
psychologists  ard  scientists  who  investigate  the  cognitive  processes  that 
underlie  problem  solving  and  reasoning  (Greeno  &  Simon,  1984) .  To  carry  out 
such  diagnoses  adequately,  it  is  necessary  first  to  do  a  task  analysis  of  the 
test  by  constructing  what  is  called  an  incidence  matrix,  which  associates  with 
each  item  an  attribute  vector.  The  latter  is  a  binary  vector  with  Is  and  Os 
as  elements,  representing  the  presence  or  absence,  respectively,  of  various 
basic-skill  attributes  in  each  item.  That  is,  if  the  jth  basic  skill  on  a 
list  is  required  for  correctly  answering  a  given  item,  then  the  jth  element  of 
the  associated  attribute  vector  is  1;  otherwise  it  is  0. 

The  determination  of  classification  groups  must  be  done  prior  to 
formulating  a  classification  space,  rule  space  which  is  defined  in  the  next 
section.  Tatsuoka  (1991)  and  Varadi  and  Tatsuoka  (1989)  successfully 
introduced  a  Boolean  algebra  algorithm  by  which  all  possible  knowledge  states 
can  be  derived  from  the  information  embedded  in  the  test  items  --more 
precisely,  from  an  incidence  matrix.  The  number  of  knowledge  states  can  often 
be  as  large  as  one  thousand. 

However,  knowledge  and  capability  states  are  impossible  to  observe 
directly  unless  we  use  computers  for  testing  and  prepare  special  computer 
programs  such  as  intelligent  tutoring  systems.  However,  developing  such 
intelligent  systems  is  very  expensive  because  they  are  essentially  domain- 
specific  and  require  extensive  programming  efforts. 

Since  only  item  scores  are  observable  in  practical  testing  situations, 
one  must  develop  a  probabilistic  method  by  which  unobservable  performances  on 
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unobservable  cognitive  tasks  can  be  inferred  from  observable  item  scores. 

One  of  the  assumptions  used  in  this  study  is  that  only  item  scores  are 
observable  and  the  underlying  cognitive  tasks  are  not  expected  to  be 
observable . 

The  purposes  of  this  study  are:  1)  To  validate  empirically  whether  or 
not  the  fraction  diagnostic  test  based  on  the  rule- space  model  is  effectively 
classifying  each  examinee  into  his/her  true  state,  and  2)  to  test  whether 
diagnostic  information  is  useful  and  efficient  for  remediation. 

A  brief  summary  rf  rule-space  model,  a  psychometrically  sound  cognitive 
diagnostic  model,  will  be  given  in  the  next  section,  and  our  procedure  for  the 
empirical  validation  of  the  model  will  be  introduced.  Finally  the  results 
from  our  study  will  be  discussed. 

Classification  Space:  Rule  Space 

A  convenient  tool  for  facilitating  error  diagnosis  is  rule  space,  which 
was  developed  by  Tatsuoka  and  her  associates  (Tatsuoka,  1983,  1985;  Tatsuoka  & 
Tatsuoka,  1987;  Tatsuoka  &  Tatsuoka,  1989).  One  of  the  upshots  of  utilizing 
the  rule-space  model  is  that  various  "misconception  groups"  are  representable 
by  ellipses  centered  on  what  might  be  called  "pure  rule  points" - -i. e. ,  points 
onto  which  are  mapped  the  response  patterns  resulting  from  the  consistent  use 
of  specific  erroneous  rules  of  operation  throughout  the  entire  test  (Tatsuoka, 
1986). 

The  formulation  of  rule  space  is  done  by  utilizing  Item  Response  Theory 
(IRT)  in  order  to  maintain  continuity  with  current  psychometric  theories. 

Item  response  functions  are  used  to  derive  an  index  f  (defined  in  the  next 
section,  Eq.7)  that  is  sensitive  to  the  unusualness  of  item  response  patterns 
(Tatsuoka,  1984;  Tatsuoka  &  Linn,  1983).  A  Cartesian  product  space  of  IRT 


ability  0  and  the  variable  f  is  used  to  formulate  a  classification  space. 

We  first  define  a  function  f(x)  that  is  proportional  to  the  coefficient 
of  regression  of  P (0)-x  on  P(0)-T(0),  so  that  when  its  value  is  close  to  zero, 
it  means  that  the  two  probability  vectors  are  almost  collinear.  Since  Pj(0) 
is  the  expectation  of  item  score  Xj  given  0 ,  the  conditional  expectation  of 
f(x)  equals  zero.  Therefore,  when  the  elements  of  an  observed  vector  x  are 
close  to  the  average  performances  on  the  test  items,  the  absolute  value  of  0 
values  will  be  nearly  zero. 

Students'  misconceptions  diagnosed  by  the  rule -space  model  can  be 
related  with  the  1RT  ability,  8.  The  unusualness  of  diagnosed  cognitive 
errors  can  be  judged  by  examining  f -values  because  the  expectation  of  f  is 
zero,  f -values  close  to  zero,  that  is,  close  to  the  0-axis,  indicate  that 
such  item  response  patterns  are  frequently  observed,  which  in  turn  means  the 
corresponding  knowledge  states  are  observed  for  many  students.  If  the  $•  -value 
associated  with  a  knowledge  state  is  large,  then  such  a  state  is  unusual.  The 
larger  a  f -value  is,  the  more  unusual  is  the  state  associated  with  this  value. 
A  similar  argument  holds  for  negative  f  values. 

By  locating  the  position  of  a  knowledge  state  of  interest  in  rule  space, 
one  can  make  two  inferences:  (1)  What  IRT-ability  level  is  likely  to  produce 
this  particular  state,  and  (2)  what  percentage  of  students  in  a  sample  will  be 
classified  into  this  state.  If  some  sources  of  error  (or  the  lack  of  specific 
knowledge)  are  shared  by  many  students,  then  the  states  involving  such  errors 
will  be  found  closer  to  0-axis  (Tatsuoka,  1990). 

Bug  distribution  The  performance  on  the  test  items  is  not  always 
perfectly  consistent  with  the  erroneous  rule  or  "bug"  (denoted  by  R)  used  most 
frequently  by  a  student,  and  the  responses  that  deviate  from  the  modal  rule 
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are  called  slips  or  random  errors.  The  item  response  patterns  deviating  by 
various  numbers  of  slips  from  a  pure  rule  pattern  R  will  form  a  cluster  around 
the  pure  rule  pattern.  We  assume  that  occurrences  of  slips  on  items  are 
independent  across  the  items.  Tatsuoka  and  Tatsuoka  (1987)  showed  that  the 
distribution  of  the  number  of  slips  is  a  compound  binomial  distribution  if  the 
slippage  probabilities  of  the  items  are  different,  and  a  binomial  distribution 
if  the  slippage  probabilities  are  the  same  across  items. 

Let  us  map  all  the  "fuzzy"  response  patterns  in  the  cluster  around  R 
into  rule  space,  by  computing  their  6  and  f  values.  Then  the  images  of  the 
"fuzzy"  response  patterns  form  a  subset  that  swarms  around  the  point  (0R,  fR) 
which  corresponds  uniquely  to  R.  The  swarm  of  mapped  "fuzzy"  points  in  the 
rule  space  follows  approximately  a  multivariate  normal  distribution  with 
centroid  (0R,  fR)  (Tatsuoka,  1990),  which  is  called  the  bug  distribution  or 
state  distribution  associated  with  R. 

When  two  sets  of  "fuzzy"  response  patterns  are  mapped  into  the  rule 
space,  one  can  apply  Bayes'  decision  rule  for  minimum  errors  to  classify  a 
student's  point  ( 6X ,  fx)  into  one  of  the  states. 

In  summary,  rule-space  model  is  a  probabilistic  model  for  cognitive 
diagnosis  and  the  model  is  applicable  to  any  domain  of  interest  at  very  low 
cost.  It  is  a  general  and  cost-effective  method  for  cognitive  diagnosis  when 
the  remediation  of  errors  is  our  concern. 

Error -Diagnostic  Adaptive  Testing  System 

Utilizing  the  rule-space  model  described  above,  Tatsuoka,  Baillie, 
Tatsuoka  (1986)  have  developed  an  adaptive  testing  system  by  which  a  students' 
sources  of  misconception  (which  produce  bugs  or  erroneous  rules  of  operation) 
can  be  diagnosed.  A  computer  program  that  sequentially  selects  an  optimal  set 
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of  items  to  administer  to  each  individual ,  and  then  terminates  the  testing  as 
soon  as  a  specified  stopping  rule  is  satisfied,  has  been  written  on  the  PLATO 
system.  The  item- sequence  selection  strategy  is  an  extension  of  the  method 
commonly  used  in  IRT-based  adaptive  testing  procedures.  The  new  system  allows 
us  to  attain  a  specified  level  of  accuracy  in  estimating  0  and  f  most  rapidly. 
The  procedure  is  an  application  of  the  theory  of  convergence  in  functional 
space ,  the  space  of  item  response  functions  in  this  case . 

Before  proceeding  to  a  description  of  our  diagnostic  testing  system,  we 
must  explain  what  0  and  f  are.  The  first  is  the  standard  person  parameter  in 
item  response  theory  (IRT) ,  commonly  characterized  as  the  latent  ability  of  an 
individual  in  the  domain  for  which  he/she  is  being  tested.  However,  it  can 
just  as  well  be  construed  as  the  individual's  achievement  level  in  that  domain 
at  the  time  of  testing;  it  is  this  interpretation  (or  definition)  of  6  that  is 
more  appropriate  for  our  purposes  here.  Either  way,  IRT  postulates  that  the 
probability  of  an  individual's  answering  a  given  item  correctly  is  a  joint 
function  of  0  and  one,  two  or  three  item  parameters.  The  latter  characterize 
the  discrimination  power  (a) ,  the  difficulty  level  (b)  and  the  "guessability" 
(c)  of  the  item--i.e.  the  chances  that  an  examinee  with  absolutely  no  ability 
or  prior  achievement  in  the  domain  will  answer  the  item  correctly.  The 
particular  functional  relation  between  the  probability  of  correct  response  and 
0,  a.  fe.  and  c  may  in  principle  be  chosen  at  will  by  the  researcher--  with 
some  reasonable  constraints  such  as  its  being  a  monotone  increasing  function 
of  0  for  fixed  &,  b,  and  c.  In  practice,  however,  only  two  functional  forms 
are  widely  used.  These  are  the  logistic  function  and  the  normal  ogive.  Here 
we  use  the  logistic  model  and  confine  ourselves  to  the  case  of  c=0,  which  is 
appropriate  when  we  are  dealing  with  open-ended  (or  "constructed- response") 
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items  as  against  multiple -choice  items.  The  two-parameter  logistic  model  has 
the  following  form: 


PjCXj-ll*)  -  1/Cl  +  exp ( - 1 . 7aj ( 0  -  bj>)]  (1) 

where  Xj  is  the  binary  score  for  item  j ,  so  that 

when  item  j  is  answered  correctly; 
otherwise 

For  short,  the  left-hand  side  of  Equation  (1)  is  often  written  as  Pj  ( 8 ),  as 
we  do  below. 

We  now  define  the  true  score  T(5)  as  the  average  of  the  Pj(0)  over  the  n 
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items : 

T(0)  -  1/nE  Pj(tf)  (2) 

Using  this  quantity  as  the  repeated  element  of  an  n- dimensional  vector,  we 
obtain 


T (0)  -  [T(0) ,T(0) .  T(*)]' 

and  hence  the  residual  (or  deviation  score)  vector 

P(tf)-T(0)  -  [PjCO-TCO,  P2(0)-T(0),...,Pn(0)-T(0)]' 

Similarly,  we  define  the  residual  vector  of  P(0)  from  the  binary-score  vector 


x: 

P (0)-x  -  [PxCO-Xx,  P2(«)-x .  Pn(*)-Xj’ 

We  then  form  the  scalar  product  of  these  two  residual  vectors,  thus: 

f*(x)  -  [P(*)-T(0)]'[P(0)-x] 

-  E  [Pj(«)-T(*)][Pj<«)-Xj]  (3) 

To  see  how  it  functions,  we  distribute  the  multiplication  over  the  second 
factor  to  get 

f#(x)  -  [P(0)-T(0)]'P(0)-[P(*)-T(0)]'x  (4) 

For  fixed  9,  the  first  term  is  a  constant.  Let  us  see  how  fe(x)  varies  with  x 
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due  to  the  second  term.  Rewriting  this  term  as 

-  E  [Pjto-KOJxj 

helps  us  make  the  following  observations:  Without  loss  of  generality,  we  may 
suppose  the  items  to  be  arranged  in  descending  order  of  magnitude  of  Pj(0). 
Then,  for  some  m,  the  first  m  terms  of  the  summation  will  have  positive 
coefficients  associated  with  the  Xj ,  and  the  remaining  n-m  terms  will  have 
negative  coefficients.  Thus,  to  the  extent  that  there  is  a  preponderance  of 
Is  among  the  scores  on  the  first  m  items  (x1,x2, .  •  •  ,2^)  and  a  preponderance  of 
Os  among  the  last  n-m  item  scores,  the  sum  (exclusive  of  the  minus  sign)  will 
have  a  large  value,  and  hence  fs(x)  will  be  small.  Conversely,  when  there  are 
many  Os  among  the  earlier  items,  f9(x)  will  be  large.  Since  the  Pj(0)  values 
are  in  descending  order  of  magnitude  [ Px ( 5 )  >  P2(0)  >  ...  >  P,,.^#)  >  Pn(0)],  a 
response  pattern  having  many  Is  among  the  earlier  items  and  many  Os  among  the 
latter  ones  may  be  regarded  as  a  "normal"  or  "typical”  response  pattern  for 
the  group  in  which  the  a_j  and  bj  values  were  estimated  (i.e.,  in  which  the 
items  were  calibrated) .  On  the  other  hand,  response  patterns  having  many  Os 
among  the  earlier  items  and  many  Is  among  the  later  items  would  be  "atypical" 
ones.  Thus  typical  response  patterns  are  associated  with  small  value  fe(x) 
while  atypical  (i.e.,  unusual)  response  patterns  get  larger  fe(x)  values. 
Hence,  f9(x)  may  be  taken  as  a  measure  of  atypicality  of  response  patterns- - 
the  larger  fg(x)  is,  the  more  atypical  the  response  pattern  is. 

The  function  fg(x)  described  above  suffices  to  serve  as  a  measure  of 
atypicality  of  response  patterns  only  for  the  population  in  which  the  items 
were  calibrated.  To  measure  the  atypicality  of  response  patterns  observed  for 
examinees  who  are  not  members  of  the  calibration  population,  we  need  to 
standardize  fg(x).  It  was  shown  by  Tatsuoka  (1985)  that  the  expectation  and 
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variance  of  ffl(x)  for  fixed  8  are 

E[ fg(x) ]  -  0  (5) 

and 

Var[f„(x)]  -  l  Pj(fl)Qj(*)(Pj(tf)  -  T (0))2  (6) 

where  Qj(0)  -  1-  Pj (8).  Thus  the  standardized  fe(x),  denoted  f,  is  given  by 

T  -  f  (0)/(var{  fs(x)  ]  )1/2  (7) 

This  constitutes  the  ordinate  of  rule  space,  the  abscissa  being  the 
estimated  value  of  8.  Thus,  each  point  in  rule  space,  which  represents  a 
particular  response  pattern  (or  an  individual  who  has  that  response  pattern) 
is  associated  with  a  coordinate  pair  (8,  f).  The  error-diagnostic  adaptive 
testing  procedure  may  be  regarded  as  a  random  walk  in  this  space,  whose  path 
is  determined  by  the  changing  values  of  the  vector  (0,  f)  as  successive  items 
are  selectively  administered  to  each  examinee  in  accordance  with  a  certain 
selection  rule. 

The  item-selection  rule  may  take  several  forms,  one  of  which  may  be 
described  as  follows.  Suppose  that,  if  an  examinee  under  consideration  were 
to  take  the  entire  test  consisting  of  n  items,  he/she  would  be  diagnosed  as 
having  a  misconception  that  places  him/her  in  misconception  group  G  in 
accordance  with  the  classification  rule  described  in  the  previous  section. 
Then,  it  stands  to  reason  that  each  new  item  to  be  administered  should  be 
chosen  in  such  a  way  that  the  examinee  would  be  hurried  toward  his/her 
ultimate  destination  as  rapidly  as  possible.  This  purpose  will  be  served- - 
even  though  the  final  destination  is  unknown- -if  successive  items  are  chosen 
so  that  the  individual's  response  pattern  is  located  as  far  as  possible  in 
rule  space  from  its  location  at  the  time  the  previous  item  was  taken.  This  is 
somewhat  analogous  to  the  method  of  steepest  descent  often  used  in  certain 
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types  of  optimization  problems. 


Once  an  examinee's  response -pattern  point  stabilizes  on  (or  converges 
to)  some  point  in  rule  space,  we  take  that  to  be  his/her  "true  point",  and  no 
further  items  are  administered.  The  next  step  is  to  determine  the 
misconception  group  to  which  that  point  most  probably  belongs.  This,  of 
course,  is  a  problem  of  statistical  classification  theory  as  described 
earlier.  Thus,  the  problem  of  error  diagnosis  is  translated  into  one  of 
statistical  classification  using  a  suitable  model.  The  model  often  chosen  for 
mathematical  convenience  is  the  normal  model--!. e.,  that  6  and  f  jointly 
follow  a  bivariate  normal  distribution  with  a  specifiable  centroid  and 
covariance  matrix  specific  to  each  misconception  group. 

Since  6  and  f(x)  are  uncorrelated,  the  covariance  matrix  becomes 
diagonal  with  1/1(6)  and  the  variance  of  f(x)  as  the  diagonal  elements 
(Tatsuoka,  1985;  1990).  The  classification  procedure  is  described  in  Tatsuoka 
&  Tatsuoka  (1987) . 


Method 

A  Task  Analysis  and  Cognitive  Attributes 

A  task  analysis  that  was  conducted  in  the  domain  of  fraction  addition 
problems  identified  15  basic  cognitive  tasks  for  carrying  out  the  addition  of 
two  fractions  of  which  there  are  three  types:  Addition  of  two  simple 
fractions,  F  +  F;  addition  of  two  mixed  numbers,  M  +  M;  and  addition  of  F  +  M 
or  M  +  F.  These  tasks  are  closely  associated  with  types  of  items  in  which 
three  positive  integers  are  used  to  form  either  a  mixed  or  fraction  number. 

The  15  basic  cognitive  tasks  are  listed  in  Table  1. 


Insert  Table  1  about  here 


However,  these  tasks  are  used  for  generating  exercise  problems  in 
instructional  units  for  remediation  and,  further,  their  mastery  patterns 
obtained  from  diagnostic  classification  are  used  for  routing  an  examinee  to 
his/her  remediation  unit(s).  Since  these  tasks  are  oriented  toward  generating 
various  types  of  items  and  focused  on  a  finer,  micro-level  analysis,  it  is 
difficult  to  see  a  close  connection  to  the  cognitive  attributes  discussed  in 
the  previous  studies  (Tatsuoka,  1984,  1986;  Birenbaum  &  Shaw,  1986).  In  order 
to  clarify  the  continuity  of  these  15  basic  tasks  with  the  attributes 
discussed  in  our  other  papers,  they  are  grouped  into  eight  categories:  {Task  1 
or  Task  2  } ,  { Tasks  1  and  2 }  ,  { Task  3  or  4  or  5  or  6 } ,  { Tasks  7  and/or  8 ) , 
(Tasks  9  and/or  10),  (Tasks  11  and/or  12  and/or  13),  (Task  14}  and  (Task  15). 
Let  us  denote  these  new  categories  as  cognitive  attributes  Ax,  through  A8. 

The  eight  attributes  characterize  38  items  with  attribute  involvement  vectors. 
Table  2  describes  the  items  by  various  combinations  of  attributes.  The  38 
addition  problems 


Insert  Table  2  about  here 


are  also  listed  in  Appendix  I  and  in  Appendix  II  with  their  parameter  values 
in  the  two-parameter  logistic  model. 

Adaptive  Testing  System  for  Cognitive  Diagnoses  in  Fraction  Addition  Problems 
An  adaptive  diagnostic  testing  system  for  fraction  addition  problems  was 
developed  on  a  computerized  instructional  system  at  the  University  of  Illinois 


10 


(PLATO  system).  The  system  is  designed  to  classify  examinees'  responses  into 
one  of  the  predetermined  misconception  groups  (called  states  of  knowledge  and 
capabilities).  For  fraction  problems,  the  misconception  groups  were 
determined  by  reviewing  the  previous  studies  (Tatsuoka,  1984;  Klein, 

Birenbaum,  Standiford  &  Tatsuoka,  1981;  Shaw  &  Tatsuoka,  1982).  Tatsuoka 
(1984)  grouped  erroneous  rules  into  several  basic  categories  by  examining 
where  each  erroneous  rule  originated.  For  example,  if  erroneous  rules  are  due 
to  lack  of  knowledge  for  making  equivalent  fractions,  then  they  are  labeled 
"errors  in  making  equivalent  fractions". 

The  15  prime  cognitive  skills  were  first  defined  as  stated  earlier  and 
33  knowledge  states  that  consist  of  various  combinations  of  the  prime 
cognitive  skills  were  selected  based  on  the  frequency  statistics  of  erroneous 
rules.  These  33  states  constituted  the  "bug  bank"  in  our  adaptive  cognitive- 
diagnostic  testing.  Paper-and-pencil  tests  that  were  given  in  the  previous 
years  were  analyzed  for  students'  errors  and  approximately  90  percent  of  the 
examinees  (N-593)  were  classified  into  one  of  the  states  (Tatsuoka,  1986). 

The  10  most  popular  knowledge  states  are: 

No. 4  Cannot  get  the  common  denominator  (CD)  but  can  do  simple  fraction 

addition  problems. 

No. 6  Cannot  get  CDs  for  the  problems  involving  mixed  number (s) . 

No. 9  Have  problems  in  simplifying  their  answers  into  the  simplest  form. 

No. 10  Mastery  state:  all  cognitive  attributes  are  correct. 

No. 11  Can  do  only  addition  of  two  simple  fractions  (F)  when  they  have 

the  same  denominators. 

No. 16  Cannot  get  CDs  and  cannot  add  two  reducible  mixed  numbers  (M) . 

Also  having  problems  with  simplification  of  the  answers. 
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No.  21 


Non-mastery  state:  all  attributes  are  wrong. 

No. 24  Cannot  add  a  mixed  number  and  a  fraction  number.  Cannot  get  CDs. 

Don't  reduce  fraction  parts  correctly  before  getting  the  common 
denominators . 

No. 25  Cannot  add  the  combinations  of  M  and  F  numbers.  Also  cannot  get 

CDs. 

No. 26  Don't  realize  that  adding  zero  to  a  non- zero  number  a  is  a  itself, 

a  +  0  =  a;  Identity  Principle. 

Table  3  describes  the  ten  most  popular  knowledge  states  by  nine 
attribute-mastery  patterns.  Appendix  III  summarizes  33  states  with  respect  to 
the  patterns  of  items  grouped  by  their  underlying  attribute  patterns. 


Insert  Table  3  about  here 


Knowledge  state  No.  10  is  the  mastery  state;  that  is,  all  answers  are 
correct  while  No  21  represents  the  state  in  which  all  answers  are  wrong. 
Remediation  Instruction  and  a  Routine  Method  based  on  Probabilities 

Fourteen  basic  skills  were  defined  and  their  instructional  units  were 
written  on  the  same  computerized  instructional  system  as  that  on  which  tests 
were  prepared.  For  example,  if  an  examinee  is  classified  into  State  No.  25, 
then  an  automated  adaptive  routing  system  sends  the  examinee  to  the  units  A3,- 
-  teaching  what  a  common  denominator  is  and  how  to  get  it-  and  Alf  -reminding 
the  student  that  F+M  type  can  be  separated  into  0  and  the  whole  number  part  of 
the  second  number  and  0+d-d  because  students  often  overlook  this  identity 
principle.  The  following  figure  shows  an  example  of  our  remediation  unit. 
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Insert  Figure  I  about  here 


The  first  box  shows  a  wrong  answer  by  a  student  and  starts  teaching  each  step 
to  reach  the  correct  answer.  If  the  student  cannot  get  the  least  common 
multiple  of  8  and  10  correctly,  then  a  specific  feedback  based  on  the 
particular  answer  will  appear  on  the  screen.  This  example  shows  that  the 
student  selected  a  common  denominator  but  did  not  choose  the  least  common 
denominator.  Therefore  an  instruction  that  teaches  multiples  of  8  and  10 
appears  on  the  screen  for  the  student.  The  box  below  in  Figure  1  indicates 
that  the  student  selected  the  right  answer,  40  and  got  the  feedback  of 
"correct!".  Then  the  computation  screen  goes  to  the  next  step,  making 
equivalent  fractions  of  1/8  and  7/10. 

The  top  of  the  remediation  instruction  in  Figure  1  shows  the  routing 
index  of  14  basic  units.  Type  11  means  item  type  11,  which  is  characterized 
by  the  attributes  Ax,  A2,  A3  and  mixed  number  addition  problems  with  different 
denominators .  Any  examinee  who  is  classified  into  one  of  the  cognitive  states 
which  includes  "cannot  do  Ax"  as  a  subset  would  be  routed  to  study  a  series  of 
instructions  indexed  by  this  label. 

However,  there  are  several  states  to  which  the  examinees  diagnosed 
"cannot  do  A1(  A2  and  A3"  belong.  Since  Boolean  algebra  defines  a  partial 
order  among  states  derived  from  a  given  incidence  matrix,  the  relationships 
among  the  cognitive  states  can  always  be  expressed  by  a  tree  such  as  the 
example  given  in  Figure  2. 
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Insert  Figure  2  about  here 


The  mastery  state  means  Group  10,  all  attributes  are  mastered.  The 
states,  "cannot  do  A*"  and  "cannot  do  A9”  mean,  respectively,  that  they  cannot 
do  A4  but  can  do  the  remaining  attributes  Alf  k2,  A3,  A5,  Ag,  A7,  A8  and  A9  and 
that  they  can  do  Ax  through  Ag  only. 

Suppose  an  examinee  was  classified  into  the  cognitive  state  "cannot  do 
A3,  A 4,  A5  and  Ag"  which  is  shown  at  the  bottom  of  the  tree  given  in  Figure  2. 
The  issue  that  arises  here  is  that  whether  Aa  should  be  remediated  first  and 
then  A5  ,  or  A5  first  and  then  A3. 

Tatsuoka  and  Tatsuoka  (1987)  introduced  what  they  called  a  bug 
distribution,  which  is  a  multivariate  normal  distribution  with  centroid  (0R, 
fR)  and  covariance  matrix, 


Bayes'  decision  rule  to  determine  whether  an  examinee's  point  X  in  the  rule 
space  belongs  to  State  A  or  to  B  is  equivalent  to  comparing  the  Mahalanobis 
distance  between  X  and  A  versus  that  between  X  and  B.  Mahalanobis  distance  in 
the  rule  space  context  is  the  same  as  considering  the  negative  loglikelihood 
ratio  of  the  two  posterior  probabilities  of  A  and  B  given  X  (Lachenbruch 
1975), 

-In  (Prob(  A  I  X  )/Prob<  B  !  X  )  (9) 

By  taking  the  position  of  State  "cannot  do  A3,  A* ,  A5  and  Ag"  as  X  and 
computing  the  two  Mahalanobis  distances,  that  between  X  and  the  State  "cannot 
do  A3,  A*  and  Ag",  would  provide  a  plausible  rule  for  a  computerized  routing 
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system.  However,  in  this  study,  the  order  for  selecting  an  attribute  from  two 
connected  states  comes  from  the  flowchart  that  was  constructed  earlier  for  a 
task  analysis  (Klein  et  al.,  1982;  Birenbaum  &  Shaw,  1986).  Attributes  A3 
(getting  the  least  common  denominator)  and  A5  (reducible  before  getting  the 
common  denominator)  can  be  located  in  the  flowchart  in  Figure  3. 


Insert  Figure  3  about  here 


It  is  now  obvious  that  As  should  be  remediated  before  A3. 

As  for  the  Mahalanobis  distances,  that  between  States  "cannot  do  A3,  Aa, 
A5  and  Ag"  and  "cannot  do  A*  A5,  Ag"  is  larger  than  that  between  States  "cannot 
do  A3,  A4,  A5  and  Ag"  and  "A3,  A4,  and  A9". 

Computational  comparison  of  these  relationships  between  the  order  from 
the  flowchart  and  the  order  based  on  Mahalanobis  distances  in  rule  space 
confirmed  that  pairs  of  attributes  closer  on  the  flowchart  have  smaller 
Mahalanobis  distances  than  do  pairs  of  attributes  farther  apart. 

The  result  suggests  that  a  potentially  good  routing  criterion  for 
remediation  is  to  compare  the  negative  loglikelihood  ratio  of  the  posterior 
probabilities  of  two  targeted  states  versus  that  of  the  state  into  which  an 
examinee  is  currently  classified. 

Data  collection:  Pretest.  Remediation  and  Posttest  in  1988  and  1989 

Three  fraction  diagnostic  tests,  pre-  and  post- tests  and  a  retention 
test  were  given  in  1988  to  students  in  the  seventh  and  eighth  grades  of  a 
junior  high  school  in  a  small  city  in  Illinois.  The  pretest  classified  each 
student  into  one  of  the  33  states.  Since  each  state  is  expressed  as  the 
mastery  and  non-mastery  of  given  cognitive  attributes,  the  examinee  was 
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assigned  to  the  instructional  units  which  teach  the  examinee  his/her  non- 
mastered  skills.  When  examinees  were  classified  into  No.  10,  the  mastery 
state,  they  did  not  participate  in  the  remediation  and  posttest  parts  but  they 
took  the  retention  test  three  months  later. 

The  examinee  was  given  a  series  of  exercises  at  the  end  of  each 
remediation  unit.  After  correctly  doing  the  exercises  at  all  the  remediation 
units  which  he/she  had  to  complete,  the  examinee  was  given  the  posttest,  which 
was  also  adaptive,  and  a  cognitive  diagnosis  was  carried  out.  The  posttest 
also  used  the  same  "bug  bank"  consisting  of  33  states. 


Insert  Figure  4  about  here 


Three  months  later,  the  retention  test  was  administered  to  the  examinees 
who  took  the  pretest  and  posttest. 

Figure  4  shows  an  example  of  an  adaptive  test  that  was  administered  to  a  real 
examinee,  whose  final  classification  was  in  State  6.  The  same  test  was  given 
to  the  examinees  three  months  later,  and  the  retention  of  examinees’  states 
was  examined. 

In  the  next  year  of  1989,  the  same  study  was  replicated  with  191 
students . 

Results 

Table  4  summarizes  the  results  of  the  six  tests  given  in  1988  and  1989. 


Insert  Table  4  about  here 


The  two  independent  studies  in  1988  and  1989  show  a  considerable 
resemblance  in  the  classification  results.  In  1988,  57  examinees  achieved  the 
mastery  state,  while  39  failed  in  all  the  cognitive  attributes  and  ended  up  in 
the  non-mastery  state  (No.  21).  In  1989,  34  were  classified  in  the  mastery 
state  and  13  were  in  the  non-mastery  state. 

State  No.  26  is  the  most  popular  knowledge  state  after  No.  10  throughout 
the  six  tests.  The  examinees  in  No.  26  are  also  very  high  achievers  and  their 
errors  are  "cannot  do  F  +  M  type  but  can  do  all  other  cognitive  attributes 
required  in  the  other  types  of  problems".  Their  erroneous  rules  are  often 
"append  1  to  F  type  and  proceed  with  all  the  computations  correctly"  (2/5  +  4 
3/7  becomes  1  2/5  +  4  3/7)  or  "  omitting  the  whole  number  part  in  the  answer" 

(  {0  +  4)  +  {2/5  +  3/7)  —  29/35  ).  The  frequencies  of  such  bugs  were  reported 
in  Tatsuoka  (1984)  and  Shaw  and  Tatsuoka  (1982). 

Insert  Table  5,  6  and  7  about  here 

However,  Table  4  does  not  show  how  the  examinees  improved  after  the 
remediation  lessons  were  given.  The  transition  of  examinees'  knowledge  states 
before  and  after  remediation  are  summarized  in  Table  5  for  the  transition 
between  pre-and  post-tests,  in  Table  6  for  that  between  posttest  and  retention 
tests,  and  in  Table  7  for  that  between  pretest  and  retention  test.  The  states 
are  listed  in  descending  order  of  the  number  of  tasks  mastered,  starting  from 
state  No.  26  (13  skills  mastered)  down  to  state  No.  21  (none  of  the  tasks 
mastered) . 

Table  5  shows  that  twenty- five  examinees  moved  from  No.  26  to  No.  10, 
(all  tasks  mastered).  Eight  from  No.  25  and  No.  16  moved  to  No.  10.  Further, 


11  from  No.  6,  7  from  No.  33  and  No.  21  moved  to  No.  10.  In  the  posttest,  89 
examinees  (39%  of  the  226  students  who  took  the  posttest)  were  classified  into 
No.  10.  Fifteen  examinees  stayed  in  No.  26  on  the  posttest.  However,  four 
examinees  who  were  in  No.  26  moved  back  to  lower-level  states.  In  all,  16  (7% 
of  N  -  226)  moved  to  lower-level  states,  but  a  majority  of  the  examinees  (93%) 
moved  to  higher-level  states.  Similar  trends  were  found  in  the  replication 
study  in  1989. 

These  changes  are  graphically  expressed  in  Figure  5.  The  points,  21, 

16,  25,  24,  33,  6,  9,  26  and  10  are  the  centroids  of  distributions  associated 
with  these  states.  The  arrows  indicate  that  examinees'  changes  in  their 
states  between  the  pre-and  posttests  are  as  indicated  in  Table  10  and  11.  The 
locations  of  33  states  in  the  rule  space  are  listed  in  Appendix  IV. 


Insert  Figure  5  about  here 


Insert  Tables  8  and  9  about  here 


As  for  changes  from  the  posttest  to  the  retention  test,  quite  a  few 
examinees  moved  back  to  their  pretest  states.  Twenty-five  examinees 
maintained  their  mastery  states  in  the  retention  test,  while  23  did  not  take 
the  retention  test.  Forty-one  examinees  (48%)  regressed  toward  lower- level 
states  from  their  posttest  state,  No.  10.  The  examinees  who  were  classified 
in  No.  26,  retained  their  skill  level  better  than  did  those  in  No.  10;  43% 
stayed  in  either  No.  10  or  No.  26  while  34%  moved  back  to  lower- level  states 
Overall,  48%  of  the  examinees  (N  -  185)  regressed  toward  lower-level  states 
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between  the  posttest  and  retention  tests.  Twenty-one  percent  of  the  examinees 
maintained  their  posttest  states,  and  31%  moved  up  to  higher-level  states  than 
the  posttest  states. 

However,  changes  between  the  pretest  and  the  retention  test  are 
encouraging,  (see  Table  7).  Fifty-five  examinees  moved  from  various  states  to 
the  mastery  state,  No.  10.  Thirty  moved  to  No.  26,  and  25  moved  to  No.  6.  If 
we  add  up  these  numbers,  110  out  of  185  examinees  (59%)  were  classified  into 
either  No.  10,  26  or  6  which  are  near-mastery  states.  However,  about  6%  of 
examinees  regressed  toward  lower -level  states  from  the  pretest  state  to  the 
retention  state. 

Since  quite  a  few  examinees  dropped  out  of  our  experiment  before  taking 
the  retention  test,  analysis  of  transition  states  lacks  in  statistical  power. 
Therefore,  the  33  states  were  grouped  into  two  categories:  those  with  serious 
vs.  non- serious  error  types.  Of  the  states  that  have  mastery  tasks,  more  than 
eight  were  categorized  as  non- serious  error  states,  and  the  remaining  states 
were  categorized  as  having  serious  errors.  Table  10  summarizes  the 
categorization  of  the  states  into  which  at  least  one  examinee  was  classified. 


Insert  Table  10  about  here 


Nine  states  were  categorized  in  the  non-serious-error  group  while  17  were 
classified  into  the  serious-error  group. 


Insert  Tables  11  and  12 


Tables  11  and  12  show  2x2  contingency  tables  of  serious  vs.  non-serious 
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error  groups  for  the  pre-  and  post- test  in  the  1988  and  1989  data.  Table  13 
is  the  corresponding  table  for  pre-test  and  retention  test  in  1988.  Tables  11 
and  12  indicate  that  diagnoses  using  the  rule- space  model  are  very  efficient. 
Indeed,  we  carefully  designed  our  remediation  instruction  so  that  if  cognitive 
diagnosis  by  the  rule -space  model  is  not  correct,  then  remediation  should  not 
work  well. 

Therefore  we  consider  the  results  shown  in  Tables  11  and  12  to  be  a  strong 
indication  of  the  reliability  of  rule -space  diagnoses. 


Insert  Tables  13 


However,  Table  13  shows  that  38  examinees  moved  from  non- serious  to 
serious  while  4  moved  from  serious  to  non- serious  error  groups.  The  number  of 
examinees  who  remained  in  non-serious  error  groups  was  110,  which  is  68%  of 
the  examinees  who  participated  in  the  retention  test. 

Proportion  Correct  for  Cognitive  Attributes 

A  unique  feature  of  the  rule- space  model  for  cognitive  diagnosis  is  that 
item-response  patterns  for  examinees  who  are  successfully  classified  can  be 
converted  into  attribute-mastery  patterns,  vhich  are  usually  unobservable. 

This  means  that  one  can  obtain  the  p-values  (proportions  correct)  for 
cognitive  attributes  as  well  as  for  items.  If  classification  rates  are  high, 
say  90%,  then  the  p-values  can  provide  researchers  with  valuable  information 
about  the  underlying  cognitive  models. 

Figure  6  shows  the  proportion  correct  scores  for  eight 
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Insert  Figure  6  about  here 


cognitive  attributes,  for  the  pretest  (N  -  287),  for  the  posttest  (N  -  225) 
and  for  the  retention  test  (N  -  185)  in  1988.  The  three  line  graphs  indicate 
that  the  proportions  correct  over  eight  attributes  for  the  retention  test 
maintain  values  higher  than  the  pretest  values  and  lower  than  the  posttest 
values.  Since  the  examinees  classified  in  the  mastery  state  (No.  10)  in  the 
pretest  (N  -  58)  were  not  routed  to  study  the  remediation  instruction,  they 
took  neither  the  posttest  nor  the  retention  test.  Hence,  the  given  proportion 
correct  scores  for  8  attributes  in  Figure  5  do  not  reflect  the  same  total 
sample  as  that  of  the  pretest,  and  they  could  be  much  higher  than  the  current 
values  if  these  mastery  examinees  would  have  studied  the  remediation 
instruction  and  took  both  the  posttest  and  retention  test. 

Time  Required  for  Completing  Remediation  Instruction 

The  average  times,  in  minutes,  for  students  in  various  states  to 
complete  the  remediation  treatment  are  summarized  in  Table  14. 


Insert  Table  14  about  here 


The  overall  average  time  for  completing  required  remediation  units  is  37.21 
minutes  across  the  knowledge  states  listed  in  Table  14.  49.8%  of  the 

examinees  reached  mastery  or  near-mastery  while  5%  of  the  examinees  remained 
in  the  serious -error  category. 


Conclusion  and  Summary 


Our  motivation  for  this  study  was  to  validate  the  results  of  cognitive 
diagnoses  by  the  rule -space  model  and  to  demonstrate  the  usefulness  of 
cognitive  diagnoses  for  instruction.  This  empirical  validation  study  strongly 
indicates  that  the  rule -space  model  can  diagnose  students'  knowledge  states 
effectively  and  remediate  the  students'  errors  quickly  with  minimum  effort. 

The  lack  of  knowledge  in  particular  domains  can  also  be  diagnosed,  and  the 
knowledge  can  be  supplied  by  instruction. 

Designing  instructional  units  for  remediation  can  be  effectively 
navigated  by  the  rule-space  model  because  the  determination  of  all  the 
possible  ideal  item- score  patterns  (universal  set  of  knowledge  states)  given 
an  incidence  matrix  is  based  on  a  tree  structure  of  cognitive  attributes, 
knowledge  states  and  items  (Tatsuoka,  1990).  Remediation  should  start  at  the 
states  whose  probability  of  mastery  for  diagnosed  deficiency  of  skills  is  as 
high  as  possible. 
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Table  1 


Fifteen  Basic  Cognitive  Tasks  required  in  the  Fraction  Addition 
Problems 


Tasks _ Description  of  Cognitive  Tasks 

1.  Separate  the  whole  number*,  a  from  the  fraction  part*,  b/c. 

2.  Separate  the  whole  number  part,  d  from  the  fraction  part,  e/f. 

3.  Get  the  common  denominator  CD,  CD  =  c  x  f. 

4.  Get  the  common  denominator  CD  when  c  is  a  multiply  of  f. 

5.  Get  the  common  denominator  CD  when  f  is  a  multiply  of  c. 

7.  Get  the  common  denominator  CD  when  f  and  c  are  factors  of  CD. 

8.  Convert  b/c  before  getting  CD,  b>c. 

9.  Convert  e/f  before  getting  CD,  e>f. 

10.  Reduce  b/c  before  getting  CD. 

11.  Reduce  e/f  before  getting  CD. 

12.  Answer  to  be  simplified,  reduce  the  fraction  part. 

13.  Answer  to  be  simplified,  convert  the  fraction  part  to  a  mixed 
number . 

14.  Add  a  whole  number  to  the  whole  number  part  of  the  answer 
after  converting  the  original  fraction  part. 

15.  Add  two  numerators. 

*  Item  a(b/c)  +  d(e/f)  ;  F  -  a  simple  fraction,  M  is  a  mixed 
number  such  as  5  3/5 


Table  2 


Description  of  Items  bv  Various  Combinations  of  Attributes 


Attributes 


a  *  0  or  d  *  0 
a  #  0  and  d  *  0 
c  #  f 

convert  before  getting  CD 
reduce  before  getting  CD 
answer  to  be  simplified 
add  two  numeratos 
adjust  a  whole  number  part 
Mixed  numbers  with  c#f 


_ items _ 

1111111111 

1234567890123456789 

1001010010011000000 

1001110010011001000 

0101100100101010010 

1110101010000000000 

1101000000000100000 

1111111010110000000 

1111111111111111111 

1001110010000001000 

0001100000001000000 


•Items  20  through  38  are  parallel  to  Items  1  through  19. 

1.  a  is  zero  if  the  first  term  is  a  fraction. 

2.  d  is  zero  if  the  second  term  is  a  fraction. 

3.  c”f  if  two  numbers  have  the  same  denominator. 

4.  If  e>f  or/and  b>f  then  the  number(s)  can  be  converted  before  addition. 

5.  If  (e,f)  *  1  or/and  (b.c)  *  1,  the  numerators  can  be  reduced  before  addition. 


6.  Answer  can  be  simplified  by  converting  or/and  reducing. 

7 .  Add  two  numerat.os . 

8.  When  a  mixed  number  answer  is  simplified  by  converting  the  fraction  part,  the  whole  number 


part  must  be  adjusted. 


Table  3 


lescription  of  the  top  ten,  the  most  popular  Knowledge  States  amom 


States  123456789 

21  000000000 

10  111111111 

6  111111110 

9  111010101 

16  110000101 

24  110101111 

25  010111101 

26  011111111 

33  100001111 

4  110111111 


11 


000101111 


labia  4 


Frequencies  of  “Students  Classified  in  Each  Class 


Groups 


1988  Data  1989  Data 

Pretest  Posttest  Retention  Pretest  Posttest  Retention 

N-287  N=226  N=185  N=191  N=175  N=171 


1 

3 

0 

1 

■1 

0 

1 

2 

2 

0 

0 

1 

0 

0 

3 

1 

1 

0 

1 

2 

0 

4 

3 

2 

7 

4 

2 

7 

5 

3 

0 

0 

1 

0 

1 

6 

26 

25 

25 

20 

16 

16 

7 

5 

5 

5 

6 

6 

3 

8 

0 

0 

1 

0 

1 

0 

9 

12 

13 

5 

10 

14 

8 

10 

57 

89 

55 

34 

69 

51 

11 

15 

2 

6 

12 

2 

2 

12 

1 

0 

0 

1 

0 

0 

13 

4 

0 

1 

3 

2 

1 

15 

0 

0 

2 

0 

0 

0 

16 

25 

7 

10 

17 

3 

13 

17 

0 

0 

1 

0 

1 

0 

18 

1 

2 

1 

1 

0 

3 

19 

4 

3 

0 

5 

1 

0 

20 

1 

0 

0 

0 

0 

0 

21 

39 

1 

2 

13 

1 

7 

23 

1 

1 

6 

5 

1 

7 

24 

9 

1 

6 

9 

2 

10 

25 

12 

6 

11 

7 

4 

6 

26 

45 

44 

30 

27 

33 

24 

(table  continues) 


1988  Data 


1989  Data 


Table  5 


Transition  Frequencies  of  Students  who  were  classified  in  the  groups  for  Pre-and-Po^t  Tests,  1988  Data 

Test  2#  of  14  13  11  11  10  10  988887766433300 

Tasks 

Test  1  Post  10  26  6  7  9  19  33  4  18  32  31  25  3  16  24  30  23  11  27  21  0  # 

Group 


#  of  Pre 
Tasks  Group 


13 

26 

25 

15 

1 

2 

11 

6 

11 

7 

4 

2 

11 

7 

1 

1 

10 

9 

3 

4 

2 

1 

2 

10 

19 

3 

9 

33 

7 

2 

8 

4 

1 

8 

18 

1 

8 

32 

1 

1 

8 

31 

3 

1 

1 

7 

25 

8 

1 

1 

7 

3 

1 

7 

5 

1 

1 

6 

16 

8 

2 

4 

1 

1 

6 

24 

3 

2 

5 

2 

1 

1 

4 

1 

1 

1 

4 

12 

1 

3 

11 

3 

3 

1 

2 

2 

1 

1  1 
1  2 

1 

1 

1 

1  1 

1 

11  2  1  1 
111  1 

1 

11  1 


1  45 

26 

5 

12 

4 

9 

1  3 

1 
2 

6 
12 

1 

3 

1  2  25 

9 

2 

3 

1 

1  15 


(table  continues) 


Test  2#  of  14  13  11  11  10  10  98  8887  7664  3  3  3  0  0 

Tasks 

Test  1  Post  10  26  6  7  9  19  33  4  18  32  31  23  3  16  24  30  23  11  27  21  0  # 

Group 

*  of  Pre 

Tasks  Group 


Tabic  6 


Transition  Frequencies  of  Students  who  were  classified  in  the  groups  for  Post  and  Retention  Tests,  1988  Data 


Test  3#  of  14  13  11  11  10  10  98888  76655  4  3  3  3  0  ? 

Tasks 

Test  2  Post  10  26  6  7  9  8  33  4  18  32  31  25  16  24  15  17  1  11  23  13  21  0  # 

Post 


*  of  Post 
Tasks  Group 


14 

10 

25 

14 

9  4  1 

2 

1 

1 

3 

1 

1 

2 

1 

1  23 

89 

13 

26 

12 

7 

7  1 

1 

2 

1 

2  1 

10 

44 

11 

6 

3 

5 

3 

2 

2 

1 

1 

1 

7 

25 

11 

7 

1 

1 

1 

2 

5 

10 

9 

2 

1 

1 

1  1 

1  1 

1 

4 

13 

10 

19 

2 

1 

3 

9 

33 

1 

1 

1 

1 

4 

8 

4 

1 

1 

2 

8 

18 

1 

1 

2 

8 

32 

1 

1 

2 

2 

2 

8 

8 

31 

2 

1  1 

2 

1 

2 

9 

7 

25 

2 

1 

3 

6 

7 

3 

1 

1 

6 

16 

1 

1 

1 

1  1 

2 

7 

6 

24 

1 

1 

4 

30 

1 

1 

3 

23 

1 

1 

3 

11 

1 

1 

2 

3  27  1  1 


(table  continues) 


Tast  3#  of  14  13  11  11  10  10  9  8  8  8  8  766554  3  3  30? 

Tasks 


Tast  2  Post  10  26  6  7  9  8  33  4  18  32  31  25  16  24  15  17  1  11  23  13  21  0  # 

Post 


#  Of 

Post 

Tasks 

Group 

0 

21 

1  1 

0 

10  1  2 

3 

1 

1 

44  62 

Table  7 


Transition  Frequencies  of  Students  who  were  classified  in  the  groups  for  Pre-and  Retention  Tests,  1988  Data 


Test  3#  of  14  13  11  11  10  10  9888876655433300 

Tasks 

Test  1  Post  10  26  6  7  9  8  33  4  18  32  31  25  16  24  15  17  1  11  23  13  21  0 

Post 
Group 


#  of  Pro 
Tasks  Group 


14 

10 

10 

1 

2 

3 

1 

1 

39* 

57 

13 

26 

16 

12 

2 

2 

1 

12 

45 

11 

6 

6 

3 

6 

1 

2 

1 

1 

6 

26 

11 

7 

1 

1 

1 

1 

1 

5 

10 

9 

4 

1 

1 

6 

12 

10 

19 

1 

1 

1 

1 

4 

9 

33 

5 

2 

2 

9 

8 

4 

1 

1 

1 

3 

8 

18 

1 

1 

8 

32 

2 

2 

8 

31 

2 

1 

1 

2 

6 

7 

25 

1 

1 

1 

1 

1 

1 

6 

12 

7 

3 

1 

1 

7 

5 

1 

1 

1 

3 

6 

16 

3 

2 

1 

1 

1 

1 

1  2 

1 

1 

2 

9 

25 

6 

24 

1 

1 

1 

1 

1 

4 

9 

5 

2 

1 

1 

1 

2 

4 

1 

1 

1 

1 

3 

(table  continues) 


last  3  #  of  14  13  11  11  10  10  9  8  8  8  8  7  6  6  5  5  4  3  3  3  0  0 

Tasks 


Tast  1  Post  10  26  6  7  9  8  33  4  18  32  31  25  16  24  15  17  1  11  23  13  21  0 


Post 

Group 


*  of 

Pro 

Tasks 

Group 

4 

12 

i 

1 

3 

11 

2 

2 

2 

1 

1 

1  1 

2 

i 

2 

15 

3 

23 

1 

1 

3 

13 

1 

1 

1 

i 

4 

3 

28 

1 

i 

1 

20 

1 

i 

0 

21 

2 

3 

5 

3  1 

5 

4 

i 

12  112 

8 

39 

55 

30 

25 

5  5 

1 

4  7 

12  4 

ii 

10  6 

2  102 

287 

39  students  who  took  Test  1  and  wre  classified  as  Group  10  did  not  take  Test  3. 


Table  8 


Percentage  of  Transition  Frequencies  of  Students  Who  are  Classified 
In  States  21.  16.  24.  25.  33.  6.  9.  26.  &  10. 


Posttest  states 


pretest 

other 

states 

10 

26 

6 

9 

33 

25 

16 

24 

11 

21 

states 

10 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

26 

57 

33 

2 

4 

0 

0 

0 

0 

0 

0 

4 

100 

6 

42 

27 

15 

8 

4 

0 

0 

0 

0 

0 

4 

100 

9 

25 

33 

17 

17 

0 

0 

0 

0 

0 

0 

8 

100 

33 

78 

0 

22 

0 

0 

0 

0 

0 

0 

0 

0 

100 

25 

68 

8 

8 

0 

0 

8 

0 

0 

0 

0 

8 

100 

16 

32 

8 

16 

4 

4 

0 

4 

0 

0 

0 

32 

100 

24 

33 

0 

22 

0 

11 

0 

11 

0 

0 

0 

23 

100 

11 

20 

20 

7 

13 

0 

0 

0 

7 

0 

0 

33 

100 

21 

18 

21 

5 

3 

0 

0 

10 

0 

3 

3 

37 

100 

Table  9 


Percentage  of  Transition  Frequencies  of  Students  Who  are  Classified 
in  States  21.  16.  24.  25.  33.  6.  9.  26.  &  10. 


Retention  test  states 


pretest 

states 

10 

26 

6 

9 

33 

25 

16 

24 

11 

21 

did  not 

take  test 

10 

18 

2 

4 

5 

2 

0 

0 

0 

0 

0 

68* 

26 

36 

27 

4 

4 

0 

0 

2 

0 

0 

0 

27 

6 

23 

12 

23 

0 

8 

0 

0 

0 

0 

0 

23 

9 

33 

8 

8 

0 

0 

0 

0 

0 

0 

0 

50 

33 

56 

0 

22 

0 

0 

0 

0 

0 

0 

0 

22 

25 

8 

8 

8 

0 

0 

8 

0 

0 

0 

0 

50 

16 

12 

8 

4 

0 

0 

4 

8 

4 

8 

0 

36 

24 

0 

1 

1 

0 

0 

1 

0 

0 

0 

0 

44 

11 

13 

13 

13 

0 

0 

7 

13 

7 

0 

0 

13 

21 

5 

8 

13 

0 

0 

13 

10 

3 

5 

5 

21 

Note .  *  39  out  of  57  examinees  who  took  the  pretest  and  were 

classified  as  No.  10  state  did  not  take  the  retention 
test.  (287  examinees  at  Pretest  but  102  dropped  out  of 
this  study  without  taking  the  retention  test) 


Table  10 


Two  Categories  of  Error  Types:  Serious  vs.  Nonserious 


Serious  Errors 

Classes  #  of  mastered  tasks 


Nonserious  Errors 
Classes  #  of  mastered  tasks 


1 

4 

8 

1 

26 

13 

2 

18 

8 

2 

6 

11 

3 

3 

7 

3 

7 

11 

4 

5 

7 

4 

9 

10 

5 

25 

7 

5 

8 

10 

6 

24 

6 

6 

19 

10 

7 

16 

6 

7 

33 

9 

8 

2 

5 

8 

31 

8 

9 

30 

4 

9 

32 

8 

10 

17 

4 

11 

1 

4 

12 

27 

3 

13 

23 

3 

14 

28 

3 

15 

11 

3 

16 

13 

3 

17  21 


0 


Table  11 

2x2  Contingency  Table  of  Seriousness  of  Error  Classes  and 
Pre-Posttests.  1988  Data 


Posttest 

Pretest  Serious  Nonserious  Total 


Serious 

21 

93 

114  (51%) 

Nonserious 

1 

107 

108  (49%) 

Total 

22 

200 

222 

(10%) 


(90%) 


Table  12 


2x2  Contingency  Table  of  Seriousness  of  Error  Classes  and 
Pre-Posttests.  1989  Data 


Pretest 

Serious 

Posttest 

Nonserious 

Total 

Serious 

19 

75 

94  (55%) 

Nonserious 

1 

77 

78  (45%) 

Total 

20 

152 

172 

(12%) 

(88%) 

Table  13 

2x2  Contingency  Table  of  Seriousness  of  Error  Classes  and 
Post-Retention  Tests.  1988  Data 


Pretest 

Serious 

Posttest 

Nonserious 

Total 

Serious 

15 

4 

19  (11%) 

Nonserious 

38 

110 

148  (89%) 

Total 

53 

114 

167 

(32%) 


(68%) 


Table  14 


Time  required  for  completing  the  remediation  instructional  units 


States 

Time  in  minutes 

2 

70.3 

5 

26.0 

6 

3.0 

7 

12.0 

9 

1.2 

11 

75.8 

13 

79.7 

16 

82.1 

18 

33.0 

19 

21.0 

21 

116.4 

25 

32.5 

26 

2.9 

31 


Appendix  I. 

38  fraction  addition  problems 


1.  2|  +  3^  = 

6  6 

17'  4  +  4  = 

33. 

M+m= 
36  36 

2  1  +  H  = 

5  8 

18'  l5  +  M  = 

34. 

W= 

3'  5 +5  = 

19  4.  +  2.= 

5  5 

35. 

2W= 

4.  2l  +  4|  = 

20.  3^  +  4^-  = 

4  4 

36. 

1+4= 

5  5 

5.  1  +  l4f  = 

2  7 

21.  |  +  1*  = 

7  12 

37. 

5-4-1  = 

6  8 

6  3!  +  4&  = 

7  7 

22.  |  +  11  = 

7  7 

38. 

£  +  1  = 

7  7 

7.  1+1= 

23.  ll  +  2f  = 

3  0 

8.  1  +  1- 

24.  1  +  2|- 

9.  4+  li2.= 

7  7 

25.  3f  +  5|  = 

i°.  |+i= 

2,  4+i- 

”•  H‘ 

27-  W  = 

12.  2|+ll  = 

28.  l|+lf  = 

13.  3^  +  22-  = 

6  4 

29.  1+1- 

i4.  u+m= 

35  35 

3°.  1  +  1- 

15.  1  +  |  = 

31.  3|+ll  = 

•  6.  if  +  f- 

32.  2l  +  3l  = 

0  0 

Appendix  II 

Values  of  Item  Parameters 

(N=595) 

item 

a  values 

b  values 

1 

.5961 

-.6818 

2 

1.3826 

.3310 

3 

.9763 

-.6910 

* 

2.0791 

.1002 

5 

1.8043 

.4495 

6 

.7118 

-.7733 

7 

1.2399 

-.5675 

8 

2.5406 

-.0420 

9 

.7030 

-.4912 

10 

1.2994 

-1.1418 

11 

2.4550 

.0477 

12 

.8223 

-.9559 

13 

2.7216 

.2354 

1* 

.8753 

-.9561 

15 

3.4782 

.0881 

16 

1.3121 

-.8332 

17 

1.1248 

-1.2200 

18 

2.2354 

.2188 

19 

1.1463 

-.9001 

20 

1.0034 

-.5024 

21 

1.8327 

.8211 

22 

1 . 1226 

-.5687 

23 

2.6170 

.1793 

2* 

2.7318 

.4305 

25 

1.1263 

-.4051 

26 

1.2935 

-.6279 

27 

5.5522 

.1568 

28 


1.1126 


-.*178 


item 

a  values 

b  values 

29 

1.4587 

-1.0376 

30 

3.2306 

.1782 

31 

1.1067 

-.6542 

32 

3 . 4223 

.3703 

33 

.8436 

-.5876 

34 

4.1896 

.1788 

35 

1.1425 

-.6935 

36 

1.1988 

-.9417 

37 

4.1909 

.2970 

38 

.8676 

-.8572 

Appendix  III 


Description  of  States  bv  Basie  Types  Riven  in  Appendix  IV 
States 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 
27 


B,  Be 


B1 

B1 

B4 

B1 

*1 


b3 

b7 

B3 


b2  b3 

B2  B3 

b5  b7 
b2  b3 

B,  Bo 


Bi  Bo 


Bi  Bo 


B*  Bi 


Bn  bI4 


Bi  Bo 


B,  Bo 


b6 

b8 

Be 


b3  b4 

B3  Bg 


Bi 


®3 

B7 


»3 

b3 

b3 

B3 


Bg  B10  B13 

b5  b6  b7 

B11  b12  b14 

b5  Bg  B7 


b4  B5  Bg  B7 


B8 

b8 


Bo  B 


10 


B11  B14 


b8  B9  B10  BU  B12  B13  B14 

b6  Bg  B10  Bi!  B12  B13  B14 


Ba  Be 


B1  B2  B3  Bg  B10 


B«  B, 


“14 


B7  Bg  Bg  B10  Bii  B12  B13  B1a 


b7 

b7 


B10 

B10  D11 


Bii  B 


14 


B1 

B1 


Bo  Bo 


b2 

b2 


Bi  Bo 


Bo  Bi 


b7 

b7 

b7 


Bq  B 


10  B11  b13  b14 


Bi  Bq 


“10 


(appendix  continues) 


States 


28  Bj  Bg  B10 

29  Bj  B2  Bg  B10 

30  B^  B2  B3  Bg 

31  Bj^  B2  B3  B4  B5  B7  Bg  B12 

32  Bj  B2  B3  B  4  Bj  Bg  B7  Bg  Bg 

33  Bj  BZ  B3  B^  B5  Bg  B7  Bg  Bg  B10 


APPENDIX  IV 


Location  of  Groups 

in  Rule  Space. 

(9.  O 

Group 

e 

r 

Standard  Error 

of  Estimates 

1 

-.666 

-.738 

.210 

2 

-.984 

.906 

.233 

3 

.245 

1.953 

.097 

4 

-.168 

-3.042 

.137 

5 

-.020 

2.876 

.112 

6 

.421 

-.296 

.114 

7 

.150 

-.028 

.096 

8 

.467 

5.221 

.122 

9 

.580 

2.147 

.144 

10 

1.818 

.507 

.778 

11 

1.184 

.165 

.249 

12 

-.833 

-.688 

.223 

13 

-1.343 

.949 

.267 

14 

-.877 

6.117 

.226 

15 

-.042 

4.574 

.115 

16 

-.639 

-.333 

.208 

17 

-.353 

4.430 

.171 

18 

.078 

1.739 

.100 

19 

.250 

1.806 

.097 

20 

-1.316 

3.670 

.264 

21 

-2.979 

.597 

.863 

22 

-1.902 

.709 

.373 

23 

-.802 

-.327 

.221 

24 

-.292 

-1.890 

.160 

25 

-.227 

-2.354 

.148 

26 

.846 

.378 

.221 

.191 


(appendix  continues) 


27 


-.483 


6.187 


33 


.305 


-1.022 


.101 


Figure  Captions 


Figure  1.  Examples  of  remediation  instruction  prepared  on  PLATO 
system. 

Figure  2 .  A  tree  representation  of  nine  cognitive  states. 

Figure  3.  A  flowchart  for  solving  fraction  addition  problems. 
Figure  4.  An  example  of  a  student's  performance  (Student  No.  48). 
Figure  5.  A  map  showing  transition  of  states  in  1988  data. 

Figure  6.  Percentage  of  correct  scores  for  eight  attributes 
described  in  Table  2. 


Question  type  11  of  1*4. 


7  8/40 


34-  +  4-L..  7  8/40  Your  anSWer  is 

°  not  correct. 

Let ’ s  do  this  pr ob 1 em  st ep - by - st ep . 

First,  add  the  whole  number  parts:  3  +  4  = 


7  ok 


fire  the  two  denominators  the  same?  n 

The  denominators  are  unequal.  They  are  8  arid  lJ0f. 

Choose  the  least  common  denominator  of  8  and-  IS':  >>  88' 
SB'  is  a  common  denominator,  but  let’s  use  40 
because  it’s  the  LEAST  common  denominator. 

The  multiples  of  8  are:  8  16  2  4  32  4.0  48  56  6  4  7  2  80 
The  multiples  of  1.0  are:  10  20  30  40  50  60  70  80 
For  the  denominator,  choose  the  SMALLEST  number 
that  is  a  multiple  of  BOTH  8  and  10. 


Quest i on  type  11  of  14. 

_  1  .7  -  a  ■, a  Your  answer  is 

3-^-  +  4t«-=  /  8/40 

d  la  not  correct . 

Let’s  do  this  problem  step-by-step. 

First,  add  the  whole  number  parts:  3  +  4  = 


7  ok 


Are  the  two  denominators  the  same?  n 

The  denominators  are  unequal.  They  are  8  and  10. 

Choose  the  least  common  denominator  of  8  and  10:  40 

Correct ! 

Now  let's  make  equ  i  va  1  ent  f r act  i  oris : 

J_  =  7  28 

8  40  10  40 


Figure  1  Examples  of  remediation  instruction  prepared  on  PLATO  system 


INPUT 

ZTZZ- 

r  tF? 

yes 


sirnpl  i  fy 


add 

numerators 

jf* - - — 

can  answer  be 
simplified  ? 

1  no  fy*s~ 

done  *3 - 


method  R 

ac+b  +  fd+e 
c  f 


I  II  I  III  I  I  IV 


equivalent 

fractions 


s i mp 1 i f s 


F  =  fraction  (b/c)  I:  LCD  prime  factoring 

M  =  mixed  number  (a  b/c)  II:  multiples  me  =  nf 

a, d  =  1st,  2nd  whole  numbers  III:  one  denom  =  multiple 

b, e  =  1st,  2nd  numerators  of  another 

c,  f  »  1st,  2nd  denominators  IV:  automatic  cf 
Method  fl:  convert  improper  fraction 
Method  B:  separate  fraction  and  whole  number  part 


Figure  3  A  flow  chart  for  solving  fraction  addition  problems 


II 


student  48:  /group 2 

17# items  done,  total  score  =  13 

dj 1=  0.1537,  grpl =  6  ;  dj2=*  2.5058,  grp2=31 


1) 

10 

1 

3/ 

5 

+ 

1/ 

5 

2) 

5 

0 

1/ 

2 

+ 

1 

10/ 

7 

3) 

25 

1 

3 

4/ 

5 

+ 

5 

3/ 

5 

4) 

27 

1 

1/ 

5 

+ 

1/ 

4 

5) 

29 

1 

4/ 

7 

+ 

1/ 

7 

6) 

32 

0 

2 

1/ 

8 

+ 

3 

5/ 

6 

7) 

21 

1 

2/ 

7 

+ 

18/12 

8) 

24 

0 

1/ 

5 

+ 

2 

5/ 

3 

9) 

2 

1 

2/ 

5 

+ 

12/ 

8 

10) 

18 

1 

4/15 

+ 

1/10 

11) 

34 

1 

1/ 

3 

+ 

4/ 

9 

12) 

13 

0 

3 

1/ 

6 

+ 

2 

3/ 

4 

13) 

37 

1 

5/ 

6 

+ 

1/ 

8 

14) 

15 

1 

1/ 

2 

+ 

3/ 

8 

15) 

8 

1 

1/ 

3 

+ 

1/ 

2 

16) 

7 

1 

3/ 

5 

+ 

7/ 

5 

17) 

9 

1 

1 

4/ 

7 

+ 

1 

12/ 

7 

*  Mahalanobis  distance  between  the  student's  position  and  the  centroid  of 
State  6  in  Rule  Space. 

**Mahal anob i s  distance  between  the  student's  position  and  the  centroid  of 
State  31  which  is  the  second  closest. 

***Items  given  . 

+  Item  scores 


Figure  4  An  example  of  a  student's  performance  (Student  No.  48) 


Unusualness  of  Response  Patterns 


Figure  5.  A  map  showing  transition  of  states  in  1 988  data 


Percentage  of  correct  scores 
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